Joint segmentation of color and depth data based on splitting and merging driven by surface fitting

This paper proposes a segmentation scheme based on the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color, geometry and surface orientation information. Normalized cuts spectral clustering is then applied in order to recursively segment the scene in two parts thus obtaining an over-segmentation. This procedure is followed by a recursive merging stage where close segments belonging to the same object are joined together. At each step of both procedures a NURBS model is fitted on the computed segments and the accuracy of the fitting is used as a measure of the plausibility that a segment represents a single surface or object. By comparing the accuracy to the one at the previous step, it is possible to determine if each splitting or merging operation leads to a better scene representation and consequently whether to perform it or not. Experimental results show how the proposed method provides an accurate and reliable segmentation

Segmentation and semantic labelling of RGBD data with convolutional neural networks and surface fitting

We present an approach for segmentation and semantic labelling of RGBD data exploiting together geometrical cues and deep learning techniques. An initial over-segmentation is performed using spectral clustering and a set of non-uniform rational B-spline surfaces is fitted on the extracted segments. Then a convolutional neural network (CNN) receives in input colour and geometry data together with surface fitting parameters. The network is made of nine convolutional stages followed by a softmax classifier and produces a vector of descriptors for each sample. In the next step, an iterative merging algorithm recombines the output of the over-segmentation into larger regions matching the various elements of the scene. The couples of adjacent segments with higher similarity according to the CNN features are candidate to be merged and the surface fitting accuracy is used to detect which couples of segments belong to the same surface. Finally, a set of labelled segments is obtained by combining the segmentation output with the descriptors from the CNN. Experimental results show how the proposed approach outperforms state-of-the-art methods and provides an accurate segmentation and labelling

Joint segmentation of color and depth data based on splitting and merging driven by surface fitting

This paper proposes a segmentation scheme based on the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color, geometry and surface orientation information. Normalized cuts spectral clustering is then applied in order to recursively segment the scene in two parts thus obtaining an over-segmentation. This procedure is followed by a recursive merging stage where close segments belonging to the same object are joined together. At each step of both procedures a NURBS model is fitted on the computed segments and the accuracy of the fitting is used as a measure of the plausibility that a segment represents a single surface or object. By comparing the accuracy to the one at the previous step, it is possible to determine if each splitting or merging operation leads to a better scene representation and consequently whether to perform it or not. Experimental results show how the proposed method provides an accurate and reliable segmentation

Segmentation of color and depth data based on surface fitting

This thesis presents novel iterative schemes for the segmentation of scenes acquired by RGB-D sensors. Both the problems of objects segmentation and of semantic segmentation (labeling) are considered. The first building block of the proposed methods is the Normalized Cuts algorithm, based on graph theory and spectral clustering techniques, that provides a segmentation exploiting both geometry and color information. A limitation is the fact that the number of segments (equivalently, the number of objects in the scene) must either be decided in advance, or requires an arbitrary threshold on the normalized cut measure to be controlled. In addition, this method tends to provide segments of similar size, while in many real world scenes the dimensions of the objects and structures are widely variable. To overcome these drawbacks, we present iterative schemes based on the approximation with parametric NURBS surfaces (Non-Uniform Rational B-Splines). The key idea is to consider the results of the surface fitting as an estimation of how good the current segmentation is. This makes it possible to build region splitting and region merging procedures, in which the fitting results are compared at each step against the previous ones, and the iterations are moved forward based on whether they turn out to be improved or not, until an optimal final solution is reached. The rationale is that, if a segment properly corresponds to an actual object in the scene, the fitting result is expected to be good, while segments that need to be subdivided or merged with other ones are expected to give a larger error. A discussion of several possible metrics to evaluate the quality of the surface fitting is presented. In all the presented schemes, the employment of NURBS surfaces approximation is a novel contribution. Subsequently, it is described how the proposed iterative schemes can be coupled with a Deep Learning classification step performed with CNNs (Convolutional Neural Networks), by introducing a measure of similarity between the elements of an initial over-segmentation. This information is used together with the surface fitting results to control the steps of a revised iterative region merging procedure. In addition, some information (fitting error, surface curvatures) resulting from the NURBS fitting on the initial over-segmentation is fed into the Convolutional Neural Networks themselves. To the best of our knowledge, this is the first work where this kind of information is used within a Deep Learning framework. Finally, the objects segmentation resulting from the region merging procedure is exploited to effectively improve the initial classification. An extensive evaluation of the proposed methods is performed, with quantitative comparison against several state-of-the-art approaches on a standard dataset. The experimental results show that the proposed schemes provide equivalent or better results with respect to the competing approaches on most of the considered scenes, both for the task of objects segmentation and for the task of semantic labeling. In particular, the optimal number of segments is automatically provided by the iterative procedures, while it must be arbitrarily set in advance on several other segmentation algorithms. Moreover, no assumption is done on the objects shape, while some competing methods are optimized for planar surfaces. This is provided by the usage of NURBS surfaces as geometric model, since they can represent both simple entities as planes, spheres, cylinders, and complex free-form shapes

Segmentation of color and depth data based on surface fitting

This thesis presents novel iterative schemes for the segmentation of scenes acquired by RGB-D sensors. Both the problems of objects segmentation and of semantic segmentation (labeling) are considered. The first building block of the proposed methods is the Normalized Cuts algorithm, based on graph theory and spectral clustering techniques, that provides a segmentation exploiting both geometry and color information. A limitation is the fact that the number of segments (equivalently, the number of objects in the scene) must either be decided in advance, or requires an arbitrary threshold on the normalized cut measure to be controlled. In addition, this method tends to provide segments of similar size, while in many real world scenes the dimensions of the objects and structures are widely variable. To overcome these drawbacks, we present iterative schemes based on the approximation with parametric NURBS surfaces (Non-Uniform Rational B-Splines). The key idea is to consider the results of the surface fitting as an estimation of how good the current segmentation is. This makes it possible to build region splitting and region merging procedures, in which the fitting results are compared at each step against the previous ones, and the iterations are moved forward based on whether they turn out to be improved or not, until an optimal final solution is reached. The rationale is that, if a segment properly corresponds to an actual object in the scene, the fitting result is expected to be good, while segments that need to be subdivided or merged with other ones are expected to give a larger error. A discussion of several possible metrics to evaluate the quality of the surface fitting is presented. In all the presented schemes, the employment of NURBS surfaces approximation is a novel contribution. Subsequently, it is described how the proposed iterative schemes can be coupled with a Deep Learning classification step performed with CNNs (Convolutional Neural Networks), by introducing a measure of similarity between the elements of an initial over-segmentation. This information is used together with the surface fitting results to control the steps of a revised iterative region merging procedure. In addition, some information (fitting error, surface curvatures) resulting from the NURBS fitting on the initial over-segmentation is fed into the Convolutional Neural Networks themselves. To the best of our knowledge, this is the first work where this kind of information is used within a Deep Learning framework. Finally, the objects segmentation resulting from the region merging procedure is exploited to effectively improve the initial classification. An extensive evaluation of the proposed methods is performed, with quantitative comparison against several state-of-the-art approaches on a standard dataset. The experimental results show that the proposed schemes provide equivalent or better results with respect to the competing approaches on most of the considered scenes, both for the task of objects segmentation and for the task of semantic labeling. In particular, the optimal number of segments is automatically provided by the iterative procedures, while it must be arbitrarily set in advance on several other segmentation algorithms. Moreover, no assumption is done on the objects shape, while some competing methods are optimized for planar surfaces. This is provided by the usage of NURBS surfaces as geometric model, since they can represent both simple entities as planes, spheres, cylinders, and complex free-form shapes.In questa tesi vengono presentati schemi iterativi per la segmentazione di scene acquisite da sensori di colore e profondità. Sia il problema della segmentazione in diversi oggetti che il problema della classificazione semantica vengono affrontati. Un primo componente dei metodi proposti è l'algoritmo Normalized Cuts, basato su teoria dei grafi e analisi spettrale, che fornisce una segmentazione basata sia sulle informazioni di colore che di geometria. Una limitazione di questo metodo è il fatto che il numero delle regioni (equivalentemente, il numero degli oggetti nella scena) deve essere deciso a priori, oppure richiede l'impostazione di una soglia arbitraria sulla metrica normalized cut per essere controllato. Inoltre, il metodo tende a restituire segmenti di dimensioni similari, mentre le scene reali spesso contengono oggetti e strutture di grandezza molto variabile. Per superare questi limiti, vengono proposti schemi iterativi basati sull'approssimazione mediante superfici parametriche NURBS (Non-Uniform Rational B-Splines). L'idea principale consiste nel considerare il risultato dell'approssimazione come una stima di quanto sia buona la segmentazione corrente. Questo rende possibile costruire procedure di tipo region splitting e region merging in cui i risultati dell'approssimazione sono confrontati ad ogni passo con i precedenti, e l'iterazione viene proseguita in base al fatto che essi risultino migliorati oppure no, fino ad ottenere un risultato ottimale. L'assunzione di fondo è che se un segmento corrisponde ad un oggetto della scena ci si aspetta che l'approssimazione mediante superfici risulti buona, mentre segmenti che devono essere ulteriormente suddivisi o uniti ad altri debbano corrispondere ad un errore maggiore. Per valutare la bontà dell'approssimazione vengono discusse diverse possibili metriche. In tutti gli schemi presentati, l'impiego dell'approssimazione mediante superfici NURBS è in particolare un contributo nuovo. In seguito, viene descritto come per gli schemi iterativi proposti possano essere proficuamente utilizzate anche le informazioni di classificazione ottenute tramite l'impiego di reti neurali convoluzionali (CNN). Infatti, in base alla classificazione viene introdotta una nozione di similarità tra gli elementi di una sovrasegmentazione iniziale, e questa informazione viene utilizzata assieme al risultato dell'approssimazione mediante superfici ottenendo una variante della procedura iterativa di tipo region merging precedentemente sviluppata. Inoltre, alcuni dati risultanti dall'approssimazione (errore, curvature delle superfici) vengono forniti in ingresso alle stesse reti neurali convoluzionali; in base alla nostra conoscenza, questo è il primo lavoro in cui dati di questo tipo vengono utilizzati in un'architettura di tipo Deep Learning. Infine, la segmentazione in oggetti ottenuta dalla procedura iterativa viene sfruttata per raffinare ulteriormente la classificazione iniziale. Viene presentata una estensiva valutazione dei metodi proposti, mediante confronto quantitativo con diversi metodi allo stato dell'arte su un dataset standard. I risultati sperimentali mostrano come gli schemi proposti ottengano risultati equivalenti o migliorati rispetto ai metodi concorrenti sulla maggior parte delle scene considerate, sia per il problema della segmentazione nei diversi oggetti che per il problema della classificazione semantica. In particolare, il numero ottimale di regioni risultanti viene automaticamente determinato dalle procedure iterative, mentre deve essere arbitrariamente deciso a priori in diversi algoritmi di segmentazione. Inoltre, non vengono poste assunzioni sulla forma degli oggetti nelle scene, a differenza di vari metodi concorrenti che sono ottimizzati per superfici planari. Questo è reso possibile dall'utilizzo delle superfici NURBS, che possono rappresentare indifferentemente sia elementi semplici come piani, sfere, cilindri che forme articolate e complesse

Scene segmentation from depth and color data driven by surface fitting

Scene segmentation is a very challenging problem for which color information alone is often not sufficient. Recently the introduction of consumer depth cameras has opened the way to novel approaches exploiting depth data. This paper proposes a novel segmentation scheme that exploits the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color and geometry information and normalized cuts spectral clustering is applied to them in order to coarsely segment the scene. Then a NURBS model is fitted on each of the computed segments. The accuracy of the fitting is used as a measure of the plausibility that the segment represents a single surface or object. Segments that do not represent a single surface are split again into smaller regions and the process is iterated until the optimal segmentation is obtained. Experimental results show how the proposed method allows to obtain an accurate and reliable scene segmentation

Joint Color and Depth Segmentation based on Region Merging and Surface Fitting

The recent introduction of consumer depth cameras has opened the way to novel segmentation approaches exploiting depth data together with the color information. This paper proposes a region merging segmentation scheme that jointly exploits the two clues. Firstly a set of multi-dimensional vectors is built considering the 3D spatial position, the surface orientation and the color data associated to each scene sample. Normalized cuts spectral clustering is applied to the obtained vectors in order to over-segment the scene into a large number of small segments. Then an iterative merging procedure is used to recombine the segments into the regions corresponding to the various objects and surfaces. The proposed algorithm tries to combine close compatible segments and uses a NURBS surface fitting scheme on the considered segments in order to understand if the regions candidate for the merging correspond to a single surface. The comparison with state-of-the-art methods shows how the proposed method provides an accurate and reliable scene segmentatio

Deep learning for 3D shape classification based on volumetric density and surface approximation clues

This paper proposes a novel approach for the classification of 3D shapes exploiting surface and volumetric clues inside a deep learning framework. The proposed algorithm uses three different data representations. The first is a set of depth maps obtained by rendering the 3D object. The second is a novel volumetric representation obtained by counting the number of filled voxels along each direction. Finally NURBS surfaces are fitted over the 3D object and surface curvature parameters are selected as the third representation. All the three data representations are fed to a multi-branch Convolutional Neural Network. Each branch processes a different data source and produces a feature vector by using convolutional layers of progressively reduced resolution. The extracted feature vectors are fed to a linear classifier that combines the outputs in order to get the final predictions. Experimental results on the ModelNet dataset show that the proposed approach is able to obtain a state-of-the-art performance